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NCLB and data-driven decision making require 
educators not only to collect assessments but 
to be able to manage them. Getting ready for 
the challenge can be the hardest part. 


C an public education suc- 
ceed in the Information Age 
without creating a culture 
where data drives decision making 
with timeliness and impact? Todays 
educators are data rich but informa- 
tion poor. The No Child Left Behind 
(NCLB) train seems to roll across the 
educational landscape like a jugger- 
naut leaving its tracks on everything 


it touches. Yet the question still 
remains, “How will a school district 
develop an intentional approach to 
data management and the selection 
of enterprise tools to support effective 
decision making?” In the course of 
this article, I will investigate some 
of the key issues educators must 
consider when they try to tackle 
this concern. 


Context for Understanding 

One thing we are very good at do- 
ing in public education is count- 
ing “things.” We can count kids, 
textbooks, and trashcans with great 
skill and precision. Often we have 

honed this to an art form even 
without advanced tools of au- 
tomation, thereby developing 
a culture of counting inputs. 

Far too often, it gets translated 
into our instructional assessment 
practices as well. Up to now, this 
practice has defined accountability 
for many educators and parents. 
In a climate of increased scrutiny 
on performance and justifying 
limited resources, the pressure 
for increased accountability 
takes on a new form and ur- 
gency for educators. One could 
easily say NCLB is not really that 
big a departure from the past ex- 
cept that it receives more media 
attention. Although this may be 
partly true, today the account- 
ability measures associated with 
NCLB are more numerous 
and more difficult to collect 
and understand. Because they are 
standards-based, they strongly focus 
on outputs instead of inputs, forcing 
us to shift the focus for our data col- 
lection strategies. Many traditional 
assessment measures simply will not 
work now. 

NCLB also calls for a new type 
of data management and reporting 
system for schools and districts. 

These tools are large in scope and 
often complex to deploy. The selec- 
tion process itself can be one of the 
most difficult parts for the project. 
The take away for a school district 
should be “this is not a simple issue,” 
and any vendor that comes to you 
claiming to have one should be 
dealt with carefully. 
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Outcomes Matter 

Once your organization decides to 
tackle this type of issue, the question 
usually becomes “what tool set should 
we use?” or “who should we partner 
with?” However, I would propose this 
is often where school districts make 
their first mistake. It is not about the 
tool or vendor. It is about you and 
what you are trying to achieve — your 
outcomes. Unless you have a struc- 
tured decision-making process, the 
adoption of any enterprise-level tool 
will become one of the most costly 
and frustrating projects you will enter 
into. In reality, you need to formal- 
ize your decision-making cycle to 
avoid what I call hiditis. Biditis is the 
commonly contagious condition in 
the public sector where voluminous 
documents are written that few un- 
derstand, few bid on, the low price 
wins, and the final product is usually 
a surprise to all involved. 

Formalize the Cycle 

How does a district begin the process 
for identifying its selection strategy 
for an enterprise tool? Prior to begin- 
ning the selection process, one of 
the most important things a district 
can do is to define its decision-cycle 
methodology. 

Many complex IT projects fail 
because they do not use a structured 
process to map out the organizational 
requirements prior to final vendor 
selection. The more complex or com- 
prehensive the application, the more 
important this step is to its successful 
implementation. 

Far too often, school districts find 
themselves in the positions of merely 
evaluating one or more products and 
trying to decide on the fly which 
one seems to be the best fit. Without 
performing due diligence to identify 
the key elements necessary for the 


software application, they are at the 
mercy of the software vendor. So 
what does a typical structured 
decision cycle look like? 

1. Define the project goals. Define 
the mission, specific need(s) 
you are trying to address, and 
desired result(s) you anticipate 
from a successful implementation. 
Ask yourself, “What are we doing 
and why is it important to do this 

■i” 

now? 

2. Define the project requirements. De- 
fine the specific requirements nec- 
essary to achieve the goals of the 
project. Ask yourself, “What must 
be done to be successful?” 

3. Define the user community. Clearly 
define who the end user will be for 
the project, both short and long 
term. Make sure to consider the 
full spectrum of users — internal 
and external. 

4. Define functional requirements. De- 
fine what functionality will achieve 
the goals. What minimum features 
must it have to meet your goals. 
What features are truly optional, 
and which are non-negotiable? 

5. Determine a master list of vendors. 
Determine the vendors who most 
closely align the functional require- 
ments and capabilities you have 
already identified. 

6. Define business and technology cri- 
teria. Identify what other factors 
need to be considered in the selec- 
tion process prior to meeting with 
vendors. For example, company 
stability, budget limits, technology 
requirements, or standards. 

7. Evaluate and select a vendor. Fully 
evaluate vendors based on your 
business and technology criteria, 
and then select one. This evalu- 
ation should include a proof of 
concept to assess and validate the 


vendor solution. This step helps 
you determine if this product is the 
one for your organization. 

Also identify to what degree each 
factor will affect the final decision. 
Will you weight the various factors? 
This understanding will be important 
because various companies will be 
strong in different areas. 

Cautions 

This model is adapted from an ap- 
proach commonly used in the private 
sector. However, we must consider 
some of the unique aspects of public 
education for it to work well for your 
organization. 

1 . Make sure to involve the right lev- 
els of stakeholders early enough in 
the process. Do not wait until the 
end. They should be involved from 
the very first step. 

2. Realize your school districts abil- 
ity to accept this level of change is 
probably much slower than what 
the vendor is ready to place upon 
you. Businesses regularly make 
large-scale change in terms of days 
or weeks. For schools, it can take 
months or years. 

3. Clearly define the difference be- 
tween a want and need. In the pri- 
vate sector, money is less of a con- 
straint if it can show a good return 
on investment (ROI). However, in 
public education that is definitely 
not the case. So we end up mak- 
ing lots of trade offs in functions 
and features. Make sure you know 
which ones are the deal breakers 
for your organization before you 
ever see any product. 

Create an Evaluation Process 

When we began the selection process 
for our data management system, we 
followed the process outlined above. 
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However there were a couple of spe- 
cific things we did differently that I 
would like to share. 

Let Us Do It. When we did the tech- 
nology testing phase (Step 7), we 
requested the final three vendors to 
install a proof of concept database 
using our data, but our engineers and 
applications team had to be the ones 
installing it under the guidance of 
their resource people. 

This was done to give our team 
a feel for how difficult it would be 
when the resource experts were not 
around later. Even though they were 
still over our shoulder watching and 
guiding, it was a much better way of 
testing the complexity of getting the 
system up and running. 

This method also gave us a way to 
collect time logs on how long various 
tasks would take for building project 
plans when we got ready for project 
implementation. If the experts did it, 
we would not be able to accurately 
assess how long each step would then 
take for our team. 

How Easy Is It — Really? Another 
technique we used in evaluation — 
which really drove the vendors cra- 
zy — ^was during the stakeholder evalu- 
ation of the product. We brought in 
the advisory team composed of vari- 
ous central office and building-level 
staff to a training lab for a full day. 

No vendors were present, and none 
was permitted to give any demonstra- 
tion of the product at this time to the 
user group. 

We had designed some basic tasks 
for everyone to try to complete with 
some minimal instruction from one 
of our staff members. We had a scor- 
ing rubric for each of the final three 
products around the same character- 
istics for each task and some other 
affective- type response questions. We 
then asked users to try to complete the 
tasks with some staff helpers assisting. 

The goal here was simple — no pun 
intended. How simple was the ap- 


plication to use? One of the critical 
factors to the effective adoption of 
any enterprise tool is ease of use. If it 
is not easy to use, users will not take 
their valuable time to learn it. So this 
environment was specifically designed 
to see how intuitive the three tools 
were with some basic instruction on 
some common tasks. By having users 
go through all three tools at the same 
time for the same tasks, they could 
quickly compare and contrast the 
relative strengths and weaknesses. 

The response from the user group 
was overwhelmingly positive to the 
experience. It also became apparent 
very quickly which products were 
easy to use and which ones were not. 
If this same thing had been done in a 
canned vendor demonstration, we are 
confident the results would not have 
been the same. 

Factors to Consider 

Once you begin the product evalu- 
ation cycle, you should use certain 
guiding principles to construct an 
evaluation rubric that you will use 
with each product solution. This pro- 
vides you a way to more objectively 
evaluate each product against the val- 
ues and features that are important to 
you instead of what the vendor is able 
to demonstrate or highlight. 

You should consider these strategic 
variables for any production solution: 

• Scalability 

• Standards-Based 

• Adaptable/Flexible 

• Supportable (technical, professional 
development, application produc- 
tion interface) 

• Empowering Users 

• Affordable (Total Cost of 
Ownership) 

The specific features or functions 
to include in your rubric come from 
the analysis you did during Step 4 
of your decision-making cycle. This 
step is important because it helps you 
focus on what was really important 


to your district or school, instead of 
what the vendor feels is important 
to you. 

Summary 

Imagine that all the dust has settled 
from the massive project manage- 
ment planning, evaluation, selection, 
and implementation. You will have 
learned a ton of lessons — some very 
positive and others that gave you a 
few more gray hairs. So wouldn’t it 
be nice to know some of them now? 
Here is my Rule of 5: 

1 . Know what you want to accom- 
plish in specific, measurable terms. 

2. Know in advance what you can 
afford to implement — short and 
long term. 

3. Know exactly who you are target- 
ing to serve with the solution. 

4. Know how you will respond 
when parents ask for access 
because they will. 

5. Know what you will do with the 
truth — the scariest of the five. 

Although we live in a time where 
accountability has become the mantra 
for public policy, we should not for- 
get we must privately hold the charge 
for ensuring that every student is suc- 
cessful in our schools. After all, we 
have been saying for decades that all 
students can learn. Maybe this will be 
our professional opportunity to step 
up and show we were right. 
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